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substantially modified, and the revised workbook is described. 
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I. Introduction 

T his paper presents the National Aeronautics and Space Administration (NASA) work performed on an 
assessment of the Technology Readiness Level (TRL) of planned Earth Science Decadal Survey Missions. It 
presents the successes and difficulties encountered in applying the TRL assessments to a variety of different 
instruments with differing levels of complexity and maturity. TRL assessments are among the many factors and 
processes employed by NASA in evaluating and selecting projects to advance from concept studies to formulation to 
implementation. The TRL assessment process aids in identifying promising new technologies that can be developed 
for possible infusion into approved space flight programs. NASA Procedural Requirement (NPR) 7120.5E, NASA 
Space Elight Program and Project Management Requirements w/Changes 1-13', requires conducting technology 
assessments at various stages throughout a program/project to provide Key Decision Point products necessary for 
transition between phases. Through the assessment process, technologies are evaluated against the TRL definitions, 
then assigned a TRL of 1 through 9. According to NPR 7120.5E, it is a good practice that technologies achieve 
TRL-6 by Preliminary Design Review (PDR). An adjunct to the TRL assessments is a Technology Development 
Plan that codifies a project’s plans for advancing the TRLs of key technologies to the TRL-6 level. Eor the 
assessments reported in this paper, the TRL definitions used were those now found in NPR 7123. IB, NASA 
Systems Engineering Processes and Requirements, Appendix E^. 

Within the NASA Science Mission Directorate, the Earth Science Division (ESD) handles the Earth sciences 
portfolio. This includes two elements which manage spaceflight missions, the Earth Systematic Mission Program 
(ESMP), and the Earth System Science Pathfinder Program (ESSP). It also includes the Earth Science Technology 
Office (ESTO) and sections for Applied Science and Research. ESMP generally handles larger, directed missions, 
such as Landsat, Ice, Cloud and Elevation Satellite (ICESat), Global Precipitation Measurement (GPM), and Gravity 
Recovery and Climate Experiment (GRACE). ESSP handles smaller, competed missions, including the Earth 
Venture series. The ESTO looks to the longer term future, by funding maturation of promising new technologies for 
future missions. Within the ESMP, a group of systems engineers led by the ESMP Systems Manager is known as the 
ESMP Systems Engineering Working Group, or ESMP SEWG. 

The ESMP formulates and executes 
missions that were described and prioritized 
in the 2007 Earth Science Decadal Survey^. 

Some of these are in their implementation 
phase, including ICESat-2 and Surface Water 
Ocean Topography (SWOT), while several 
others are in pre-formulation. This paper 
concerns work done in 2013 on three of these 
pre-formulation missions. Each of the three 
mission teams had been instructed in a yearly 
letter, in October 2012, from NASA ESD to 
the program scientist and program executive 
to do a technology readiness assessment. The 
three missions are Aerosol-Cloud-Ecosystems 
(ACE), Active Sensing of CO 2 Emissions 
Over Nights, Days and Seasons (ASCENDS), 
and Geostationary Coastal and Air Pollution 
Events (GEO-CAPE). Each team was directed 
to provide an assessment of the current TRL 
for the instrument. ESTO and the ESM 
SEWG were to assist them in that process and 
provide and independent review of the 
instrument assessments. The fourteen 
instruments are listed in Table 1. 

Many of the instrument concepts in pre- 
formulation started out as low-TRL efforts, 
which advanced under ESTO investment and management. ESTO fosters the development and assessment of 
innovative remote-sensing concepts in ground, aircraft, or engineering model demonstrations. Under NASA 
Research Announcements (NRAs), ESTO competitively selects proposals to develop component, instrument, and 
information technologies. TRL levels typically range from 2 to 4, with the expectation that after 2 to 3 years the 


Mission 

Table 1. Instruments Assessed 

Instrument Name Institution 

GEO-CAPE 

COEDI 

GSEC 

GEO-CAPE 

CHRONOS 

LaRC 


ACE 

Dual frequency radar 

JPL 

ACE 

ACERAD/C2D2 

GSEC & NGES 

ACE 

ACE Lidar 

LaRC 

ACE 

Pol MPSI 

JPL 

ACE 

APS Polarimeter 

GISS 

ACE 

PACS Polarimeter 

UMBC 

ACE 

Ocean Color Multi-channel 

GSEC 


spectrometer 



ASCENDS 

LMCT Lidar / CO 2 LAS 

JPL 

ASCENDS 

Sounder with CO 2 channel 

GSEC 

ASCENDS 

Broadband CO 2 Lidar 

GSEC 

ASCENDS 

ITT Lidar 

LaRC 

ASCENDS 

2 micron pulsed Lidar 

LaRC 
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TRL of the proposed development will advance by at least one level. The projects are actively managed by ESTO, 
and required to have a detailed work plan and budget that is tracked using regular technical and financial reporting 
and reviews at 6-month intervals from the start date of the project. 

The TRL workbook for these ESTO-funded projects was developed to aid the Principal Investigators (Pis) in 
assessing the technical progression of their projects. The PI is required to have the first iteration of the workbook 
completed by the first annual review of their project. In preparing for this review the PI iterates inputs to the 
workbook with their assigned ESTO technology development manager to ensure the TRL rating is both objective 
and adequately substantiated. 

A process for conducting a Technology Assessment is provided in the NASA Systems Engineering (SE) 
Handbook'*, Appendix G, and the initial part is called a Technology Maturation Assessment (TMA). This is to be 
performed using the TRL scale, or if another scale is used, to be mapped into the TRL scale. Eor the purpose of this 
paper, “TRL assessment” will be considered to be equivalent to a TMA. 

Early TRL assessments on NASA projects are typically performed by mission advocates, and then presented to 
NASA management along with other data to provide a complete picture of a proposed mission. These assessments 
can vary significantly due to differing interpretations of the TRL definitions and due to advocate and non -advocate 
perspectives. In 2011, NASA’s ESD recognized the difficulties inherent in weighing the TRL assessment of one 
prospective mission against another. Significant inconsistencies were noted among the TRL assessments made by 
instrument teams in the pre-formulation stages of the program. The TRL determinations had been performed using 
informal processes and peer reviews, and the definitions were being inconsistently applied. 


II. Evaluation Process 

In November 2012, the ESMP SEWG, at the request of ESD, embarked on a process aimed at improving the 
consistency and uniformity in performing TRL assessments. The ESMP SEWG elected to adapt the TRL 
Workbook^ that ESTO had developed. The SEWG and ESTO teams met to review the TRL workbook, and found it 
to be practical and straightforward to use on pre -formulations mission assessments, with the exception of needing 
adjustments to fit the need of ESMP missions. It was a Microsoft ExceF*^ workbook, and is partially illustrated in 
Eigure 1. 

The TRL Assessment Worksheet used for these assessments itemized the parts of a system and the rationale for 
the TRL assigned to the components, assembly and the overall system. The process was used to ensure all parts were 
accounted for, and highlight those needing further efforts to improve their readiness for use. The TRL Workbook 
was composed of a set of worksheets, which appear in tabs. Most of these are visible in the bottom of Eigure 1. The 
nine tabs are as follows: 

Background and version info - a brief background and top-level instructions for completion of the worksheet as 
well as a record of workbook version as a form of configuration management. 

TRL Definitions - TRL definitions reproduced directly from NPR 7120.8, NASA Space Elight Research and 
Technology Program and Project Management Requirements®, App. J, but now found in NPR 7123. IB, Appendix E. 

Other Definitions - definitions of common terms used in the TRL definitions. These definitions were 
reproduced from NPR 7120.8, Appendix J. 

TRL Worksheet - the core of the TRL assessment workbook as illustrated in Eigure 1 ; all other tabs support 
the completion of this worksheet. 

Component Questions - questions to be answered for each component in the Product Breakdown Structure 
identified as a technology item in the affirmative in order for the component to attain the relevant TRL. 

Assembly Questions - questions to be answered for each assembly in the Product Breakdown Structure 
identified as a technology item in the affirmative in order for the assembly to attain the relevant TRL. 

System-Subsystem Questions - questions to be answered for each system/subsystem in the Product 
Breakdown Structure identified as a technology item in the affirmative in order for the system/subsystem to attain 
the relevant TRL. 

Mission TRL Assessment - explicit set of instructions for completing the TRL worksheet. 

INFLAME Example - an example of a completed worksheet, see Eigure 2. 
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Figure 1. TRL Worksheet from TRL Workbook. This blank worksheet is filled in with the product breakdown, 
technology level assessment, and implementation approach. 


The following describes the categories of data that are to be documented in the TRL workbook. 

The Product Breakdown Structure is similar to a Master Equipment List (MEL). If a MEL exists, it can serve 
as the starting point for the Product Breakdown. The instrument is decomposed into its subsystems, the assemblies 
within each subsystem, and the components within those assemblies. Each instrument subsystem, assembly, and 
component is entered into a separate row. Some examples of instrument subsystems would include the optics, the 
structure, the main electronics, thermal control, command and data handling, and power. An assembly is defined as 
a distinct box. For example on an instrument, it might be a separate optics module, electronics module, or a pointing 
platform. A simple instrument might not have any assemblies if it is a single box. A component is defined as any 
separate element within an assembly. For an instrument, a component could be a detector, cryo-cooler, aluminum 
mirror, etc. This is done for the hardware, but the workbook does not call for a similar treatment of software. 

The Target Environment is the expected operational environment. For satellites it is the orbital parameters, 
such as altitude and inclination, but could be something else for suborbital, airborne or balloon systems. The 
Relevant Environment is the critical environment identified for the operational environment, that an element or the 
entire system must be tested against to gain TRL 5 or 6 level of maturity. 

For the components, assemblies or subsystems, any Key Technology items are identified by writing a short 
phrase defining the new technology, i.e., developments that have not been accomplished before. This should include 
any items that are beyond the limits of routine engineering and that might require some form of verification that the 
item designed performs in the manner consistent with the intended purpose. Technology items are most often 
identified at the component level. However, if the new effort is to integrate the components in a new way, or to new 
requirements, then the Technology Item is the novel integration effort at the assembly or higher level. 

The Workbook contains separate tabs with a series of Questions that, when answered, will guide the assessor to 
the appropriate TRL level for that item. For each item (component, assembly, or subsystem) all questions must be 
answered in the affirmative in order to claim the corresponding TRL level. For items that are truly NOT new 
technology (i.e., are routine, proven engineering) the TRL assessment entry is simply “No new technology.” This 
should not be used if the component is used in a way that differs significantly from past, proven applications. Note 
that while there is a version of the workbook that has a second TRL column for software, the one used in this study 
does not. Software could be included by making one or more lines for it. 
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Assembly 
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Technology 
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TRL 

Justification 

Implementatio 
n Method 

Vendors (if applicable) 

INFLAME 

Instrument 




3 

Longsiave measurements belov 1100 cm' 
‘ are at TRL 7 as a result of the flight 
demonstration, but improvements to the 
LW and SW instruments are required to 
achieve the full wavelength coverage of 
100-0.3 pm. 

Designed, 
assembled, and 
tested in 
house. 


Target 

Environment 

Airborne 
Instrument 
for use on a 
Lear let at 
45,000 ft. 















LongKttve 

spectrometer 




6 

Flight spectra meeting requirements 
over part of the desired w avelength 
range have been obtained. 

Designed, 
assembled, and 
tested in house. 



ODtomechanical 



6 

Flight spectra meeting requirements over 
part of the desired wavelength range have 
been obtained. 

Designed, 
assembled, and 
tested in house. 




Fourier 


6 

Demonstrated direct measurements of net 
flux during test flight. 

Designed, 
assembled, and 
tested in house. 


transform 

soectrometer 

(FTS) 




Bilayer pellicle 
beamsplitter 

6 

Full wavelength coverage demonstrated in 
lab calibration 

Designed in 
house; 
fabricated 
extemallv. 

Smithsonian Astrophysical 
Observatory. 







Designed in 



I ► ►!_ Assembly Questions System-Subsystem Questions Mission TRL Assessment INFLAME Example J i I 

Figure 2. INFLAME Example Worksheet from TRL Workbook. This worksheet illustrates how to follow 
the instructions and complete a TRL evaluation using the workbook. 


The TRL of a System cannot be higher than the lowest TRL of its constituents. At higher levels of assembly, the 
questions in the tabs for Subsystem/System ask if the prototype has been tested, and if these questions could not be 
answered in the affirmative, then a higher level of assembly may have a lower TRL than its constituents. 

Justifications are to be provided to substantiate the TRL selected in each case. The justifications should be in the 
form of analysis and test reports, whitepapers, log entries, etc. Note that in practice, most instrument developers in 
this study provided background information and details of their overall test program, descriptions of the various 
developmental models (brassboard, engineering model, etc.), their strawman space-flight instrument design, and the 
outcomes of the various tests in a presentation format, in parallel with the spreadsheet. The spreadsheets did not 
typically include formal references to test reports, as one might find in formal flight test verification documentation. 
The “Justifications” were informal descriptions of the basis for the TRL rating. 

The Implementation Approach section aids in the overall assessment process and to determine that a 
maturation plan exists for the lower TRL components. The assessor is expected to enter information that will enable 
a reviewer to determine the relative maturity of the implementation approach. The assessor should enter the 
implementation method including technology maturation plans. Questions to be addressed include the following: 

• “Who is expected to implement the technology?” 

• “Has a vendor been selected and are they under contract?” 

• “What is the plan for advancing the technology?” and “Is there a documented plan for the advancement?” 

In addition to producing the updated TRL Workbook described above, the SEWG also released the TRL 
workbook instructions as a configuration control board controlled document, GSFC 420-01-05, ESMP TRL 
Assessment Process^. Note that significant work on the TRL assessments was already underway before this 
document was signed off. 

Several of the instrument teams were already engaged with the ESTO, and had applied for and won technology 
development funds from ESTO. They had previously used the ESTO TRL workbook to perform a self-evaluation, 
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and simply had to update it to reflect their current status. Other teams had never used this assessment method, so the 
SEWG and ESTO teams provided the necessary guidance. The original direction had been issued to the Program 
Scientist and Program Executive of the three Decadal Survey Missions, ACE/ ASCENDS/GEO-CAPE, to perform 
the assessments with the SEWG. The SEWG/ESTO team began their 2013 TRL assessment process by holding a 
kick-off meeting in Eebruary 2013 with the teams and the Program Scientist or Executive. The teams were then 
provided a copy of the current version of the TRL Assessment workbook with advice that this was to be used to 
complete the work requested in the NASA ESD annual letters. Consultation was provided by assigning a SEWG 
team member familiar with the instrument and the newly modified TRL assessment workbook and process to work 
with the instrument team to assist in completing the workbook, answering questions, and checking it at intermediate 
points. 

Once each instrument team had completed their assessment, members of the team met with the SEWG to review 
the results. These meetings were held between May - July 2013. The meetings were conducted in-person at the 
NASA centers closest to the home institutions of the instrument teams, in Greenbelt, MD, Hampton, VA, and 
Pasadena, CA. The SEWG/ESTO reviewers provided informal feedback at each of the meetings to the instrument 
teams. Eeedback letters made recommendations as to specific changes that should be made, and recommended that a 
revision to the assessment be made. In September 2013, the Program Scientists and Program Executives reported to 
the NASA Earth Science Division on progress on their Decadal Survey missions. These reports included the results 
of the TRL assessments performed with the SEWG. After completing these assessments, the SEWG/ESTO team met 
to discuss the outcome of the activity, and compiled a set of findings and recommendations. These are listed in the 
following section. 


III. Findings and Recommendations 

A. Finding 1. Utility of the Workbook. 

Properly used, the workbook provides a good way of carrying out the NASA instructions on performing a TRL 
assessment. If the instructions are carefully followed, including answering all the questions in the tabs for the 
various levels of assembly, including the availability and testing of analytical, sub -scale, and full-scale prototypes, 
an assessment that is compatible with NPRs and the NASA SE Handbook will result. 

B. Finding 2. Evaluation Team Membership 

The evaluation team members should match the needs of the assessment. For the purpose of these assessments, a 
team of a SEWG representative from each center plus an ESTO representative was sufficient. For a more in-depth 
evaluation, technical experts on the relevant technologies should be added. Similarly, the level of independence 
needed depends on the potential impact of the assessments. For the pre -formulation stage, a self-assessment 
followed by an evaluation by a small panel is adequate. For a higher stakes situation, such as a TRL 6 determination 
in a high budget space mission, an independent assessment is called for. 

C. Finding 3. Presentation with Background Material 

The completed workbook must be accompanied by a presentation explaining the instrument and the development 
accomplished so far. The TRL workbook alone proved to be insufficient with the early reviews, and it became 
apparent that additional information was needed to aid in the understanding not only of the technology items, but of 
the system architecture the technology would eventually be infused into. Teams who were reviewed later in the year 
were asked to prepare a presentation of background material, with key information such as the objectives and 
requirements for the instrument, a product breakdown functional diagram, prior work, current status, future plans, in 
addition to filling out the workbook per the instructions. The additional supplemental material served to increase the 
review team’s understanding of an unfamiliar instrument concept and helped greatly with the technology 
assessments and technology infusion. 

D. Finding 4. Not a Design Review 

A Technology Readiness Assessment is not a Design Review. In a couple of cases the SEWG reviewers became 
overly interested in the novel instrument design concepts being presented as background information, resulting in 
questions more often concerning the design and aimed at solving the engineering problems of the concept design. 
This diluted the proper focus on the technology maturity aspects of TRL assessment. Future reviews will still 
require the instrument teams to provide and present the supplementary material along with the results in the 
workbook. However, the review team will have to be diligent at focusing on the maturation questions contained 
within the workbook. The review team must ensure that the TRL evaluation process has been completed properly, 
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and the team must guarantee each of the technology items and their roll-ups have sufficiently met their maturation 
criteria. 


E. Finding 5. No New Technology 

The instruction to identify portions of the system that are mature as “No new technology’’ can save time and 
focus attention. The detail and level of completeness at filling out the TRL worksheet varied greatly. In one case, 
great effort was given to filling out the TRL workbook. Each and every level of the instrument product breakdown 
had detailed maturity assessments. Even for the parts of the instrument design that contain no new technology 
items, assessments were performed. The end result was a very thorough and detailed instrument TRL assessment 
with a completely filled in workbook. While a complete or comprehensive assessment may be necessary for a 
proposal or cost estimate, it is not necessary for a technology readiness assessment. In another example, one 
instrument team filled in only sections of the TRL workbook that contained new technology items. The other 
sections of the instrument architecture that contained no new technologies items were either left out or were filled 
with a “NT” (not technology). In this particular case, it was extremely difficult to gain a complete understanding of 
the instrument maturity and left a feeling of large uncertainty with the review team. Both cases represent extreme 
examples of the amount of effort at filling out the worksheet. The solution lies between the extremes. In cases where 
there are subsystems and assemblies with no new technologies, it is still beneficial to the stakeholders and decision 
makers to know the maturity levels of those subsystems and assemblies. If a strong justification can be provided for 
assigning a high TRL at a high level of assembly, then details at lower levels of assembly need not be provided. An 
efficient approach is to start with a complete Mission Equipment List (MEL), identify a TRL for each (with minimal 
explanation), then focus on the new technology in the TRL workbook. This allows the reviewer to see that adequate 
consideration was given to the mature items, without needing to see it in the format of the TRL workbook. 

F. Finding 6. Software TRL 

The SEWG/ESTO team found that the instrument developers did not generally accept the software TRL 
definitions as valid or useful. Those that did attempt to rate their software made various errors. One team rated very 
standard, commonplace, and relatively simple flight software as the lowest TRL of the entire system because that 
part of the flight software had not yet been started. Another team rated ground data processing software for higher 
level data products as being at a low TRL. Since this software does not need to be ready early on, and may even be 
done by a separate group under a separate agreement, and will not likely be very expensive or hold up the flight 
system development, the SEWG/ESTO team advised that this not limit the system’s TRL. While the SEWG/ESTO 
team considered several possible cases, none were found for which a software TRL rating distinct from that of the 
flight hardware seemed to be meaningful. 

G. Finding 7. Need for Improvement of the Workbook 

The Questions in the tabs of the workbook seemed to be difficult for developers to follow consistently. While 
some of the instrument development teams used the questions, during the review it was somewhat awkward to turn 
back to the tabs and check on whether all questions were answered. The tendency was for the reviewers to rely on 
their recollection of the questions. It would be beneficial for the reviewers to select a portion of the workbook, and 
verbally ask for confirmation that the answers to the questions were ‘yes’ or ‘no’ for a selected set of TRL ratings. 
In preparation for this, the questions must be confirmed to be self-consistent, logical, aligned with the TRL 
definitions, and value-added. The SEWG team considered adding columns to the workbook for reviewers to affirm 
each question, but it was not clear how to implement this in a simple spreadsheet. Discusions of how to resolve this 
issue led the SEWG to look back at the source documents, and consider more substantial modifications to the 
workbook. 

H. Finding 8. Reporting on Evaluations 

It is desirable for the group that is evaluating the assessments performed by the development team to report back 
directly to the official who made the request. The development team may not respond to the recommendations, and 
the official needs to know the result of the group’s evaluation. Without this, there may be little or no value to the 
evaluation because it is ignored. 

I. Finding 9. Roll-Up 

Roll up using the weakest link approach, but allow exceptions. When to assign a higher level of assembly a 
lower TRL than the lowest of its constituents is a difficult and contentious matter, and the tendency is to take the 
simple ‘weakest link’ approaeh, whieh produces the highest possible TRL for a system. However, examples can be 
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found, such as the co-alignment of otherwise mature hardware items under extreme environments, such that the 
feasibility of the system to achieve the required performance is in doubt. 

J. Recommendation 1. Continue to use the TRL Workbook. 

Continue to perform the Technology Maturity Assessments using the TRL Workbook directly. The SEWG’s 
workbook is similar to other approaches for determining technology maturity. The SEWG’s approach uses a 
Microsoft ExceRM workbook and implements the approach recommended in the NASA SE Handbook. It is 
consistent with NASA’s TRL definitions in NPR 7123. IB. 

K. Recommendation 2. Research Alternative Approaches 

Do additional background research on other methodologies and approaches, especially on the NASA guiding 
documents, to guide the updates to the TRL Workbook. This research was performed, and is summarized in Section 
IV. 

L. Recommendation 3. Update the Workbook. 

Review differences between the NASA SE Handbook and the TRL Workbook. Evaluate what modifications 
should be made to the workbook to improve it. Improve the clarity of instructions to avoid confusion. The outcome 
of this work, informed by the review of methodologis and approaches in Section IV, is described in Section V. 

M. Recommendation 4. Special Case - Electronics 

There are certain special cases which are worthy of specific guidance. One of these is electronics. For 
electronics, a convention has been adopted at the GSFC which the SEWG team finds very useful. If an electronics 
box has been determined to be able to be produced with existing parts that are flight -qualified for the relevant 
environment (including radiation), using conventional assembly methods, then it can be assigned a TRL of 6 for the 
application in question, and designated “No New Technology”. There is still development risk, but it is not 
technology risk. 

N. Recommendation 5. Clarify New Technology vs Advanced Engineering vs Heritage. 

During the TRL evaluations that were performed in 2013 by the SEWG, some confusion regarding what was 
considered “new technology” vs “no new technology” arose in several of the reviews. To address those concerns, 
changes within the workbook are now required identifying each assembly, subsystem, and system as new 
technology, advanced engineering, standard engineering, or heritage along with a justification for each of those 
assignments. The current process of identify “new technology” vs “no new technology” does not adequately capture 
the issue for those engineering developments that push the envelope of engineering bordering on the edge of 
technology development. This is future work for the SEWG, and has not yet been addressed in the TRL Workbook. 

IV. Review of Methodologies and Approaches 

The SEWG conducted a literature search to review other approaches to technology readiness level evaluation 
and other supplemental readiness evaluations to provide additional information to enhance the understanding of 
TRL. Those other supplemental assessment approaches included: technology readiness level, system readiness 
level, the difficulty associated with advancing a technology to higher levels, and the determination of programmatic 
risk related to the technology development. 

A. Technology Readiness Levels and Assessment Methodology 

A NASA researcher, Stan Sadin, conceived the technology readiness methodology in 1974 and later formally 
defined technology maturities in seven distinct readiness levels. Later in the 1990s, the readiness levels would 
increase to nine levels to include technology development through operations and support. J. Mankins, the former 
Director of the Advanced Concepts Office at NASA Headquarters, expanded the TRL descriptions in Ref. 8, later 
becoming a NASA Standard in NASA Policy Documents. These definitions were widely accepted and have since 
been adopted by many agencies of the US government including the Department of Defense (DOD) and Department 
of Energy (DOE) as well as the European Space Agency (ESA). It was concluded that the NASA TRL metric has 
become a standard across other US Agencies and ESA and therefore has a high degree of acceptance. 

NASA, DOD, and others have used various tools for determining technology maturity. Each of the tools 
surveyed calculate a single value TRL based on allowing the user to answer a series of question about the 
technology. Many of the tools are Microsoft ExceF'^ spreadsheet application based having the questions embedded 
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within the spreadsheet. Once the questions have been answered, the tool guides the user to the achieved TRL. The 
advantage of the tool is that several technologies can be evaluated with the same standard questions and the tool 
provides a repeatable process for each TRL assessment. 

Other suggestions have been put forward to supplement the TRL definition for purposes of assessing the level of 
technology readiness of the system into which the technology will be incorporated. These suggestions take into 
account the difficulty of technology integration with the different elements of the system architecture called 
integration readiness level. The integration readiness level thus becomes a factor along with the technology maturity 
level for determining the system maturity level. There are various such meta-scales, combining TRL with other 
readiness factors to create a new, more comprehensive scale. The NASA SEWG is evaluating these metrics for 
inclusion in the process. 

For the TRL evaluation performed by SEWG these metrics were not included. The SEWG evaluation depended 
on the experience of the team members to take these factors into account. 

The normal development of flight hardware for NASA is challenging. Many of the elements of a spaceflight 
mission have such stringent requirements that there is always some development risk, even if every part of the 
system has direct flight heritage. There can be supply chain problems, loss of key personnel, and a company may 
stop production or move facilities or sell part of a company. Some components are so difficult to make that almost 
anything could result in failure of a previously reliable component. A high TRL is therefore not a guarantee of 
success, but it is a way to reduce Technology Development Risk. 

As a result, NASA tends to separate engineering development risks from technology development risks. 
Experience shows that large development problems mostly arise when flight system development is dependent on a 
technology development schedule and budget compliance. The notion is that if you can reduce the technology 
readiness risk early, a competent team can handle the rest of the development risks with a reasonably high level of 
assurance, and with a quantifiable chance of modestly higher costs and delayed schedules. So, for NASA, it seems 
inappropriate to try to re-combine development risk with technology risk into a single metric, since they have such 
distinctly different effects on flight projects. 

The difficulty in applying the TRL scale to software is explored by Seablom in Ref. 9. He describes the general 
value of using the TRL scale, and how it has been used to evaluate Earth science missions and technology 
developments. He notes that the software definitions are only recently added, and there is a limited body of 
experience relating them to outcomes on NASA missions. He notes specific difficulties with the current software 
TRL definitions, regarding how they differ from criteria used by the Carnegie-Mellon Software Engineering 
Institute. 

B. NASA Requirements 

NASA has two primary documents that concern requirements for technology development. The first is NPR 
7120.8, which guides technology development (TD) projects. This document says that a technology assessment and 
development plan is central to the TD project. For NASA’s Earth science portfolio, this early technology 
development is handled in a number of ways. Some is being done by industry or other agencies using their own 
resources, and NASA is a by-stander. Some is funded and managed by ESTO, generally guided by the priorities of 
the ESD. Some is funded by NASA center-level chief technologists, or by the NASA Office of Chief Technologist 
via the Space Technology Mission Directorate. In at least two other NASA programs in the area of Astrophysics, 
technology money is prioritized by the program office and dispersed by the Astrophysics Science Division of SMD. 
For nearly all of these cases, however, the general approach is to assess the current TRL, and then assess the scope 
of the work that remains to be done to achieve higher levels of maturity. 

NPR 7120.5E calls for a document to be released at the end of Pre -Phase A, at the time of the Mission Concept 
Review, known as the Formulation Agreement (FA). It calls for the FA to include an appendix that describes the 
necessary technology development, and the NPR calls for a Technology Development Plan (TDP) to be baselined at 
this time either as a stand-alone document or in the form of the FA appendix. It requires the following: 

Identify the specific new technologies (Technology Readiness Levels (TRL) less than 6) that are part of this project or 
single-project program; their criticality to the project’s or single-project program’s objectives, goals, and success criteria; 
and the current status of each planned technology development, including TRL and associated risks. Describe the specific 
activities and risk mitigation plans, the responsible organizations, models, and key tests to ensure that the technology 
maturity reaches TRL 6 by PDR. 

A TDP describes the technology advancement needed, with the steps necessary to bring the technology to a level 
of maturity for successful integration into a program/project. The plan provides the roadmap and the amount of 
resources and effort needed for technology development and infusion. The TDP includes the results of a risk 
assessment. The NPR requires that the status against this plan be reviewed at subsequent gate reviews up through 
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PDR. This approach achieves the objectives of the other analysis approaches found in the literature, and is 
compatible with the TRL evaluation discussed in this paper. 

The NASA approach, which is addressed in the NASA SE Handbook, enables the assessment of technology 
maturity at all levels of assembly. It addresses how to incorporate a new technology at the component level into an 
otherwise mature system configuration. It also allows consideration of challenges in combining existing elements 
under more stringent requirements for performance, or use of resources (e.g., mass), or environments (thermal, 
radiation) by assessing each sub element for technology readiness and building up the system by evaluating the TRL 
for sub-systems and the TRL of combinations of sub-systems at the system level. The integration is inherent in the 
process and has usually been achieved by an integrated test of the sub-systems at a lower level conducted in an 
equivalent environment. An example of this is the aircraft flight of an instrument under evaluation. The aircraft 
flight has been accomplished with the sub-systems of the instrument system; however, the results must be scaled for 
the intended environment and configuration of the instrument. 

The SEWG/ESTO team concluded from this research, as well as the prior findings and recommendations, that 
re-working the workbook to align closely with the NASA SE Handbook was needed. 

V. Modifications to the Workbook 

After conducting literature searches, investigating numerous other approaches to TRL evalutions, and the 
lessons learned from the 2013 SEWG TRL assessesment, the SEWG decided to continue to use Microsoft Excel™ 
spreadsheet application, however with major modifications to the workbook. The SEWG decided to migrate away 
from having the technologist answer a series of questions about the new technology(s) to assess readiness level to a 
different approach based on the notional example described in NASA SP-6105 Rev 1. This approach reqires the 
technologist to fill in a TRL assessment table very similar to the TRL assessment matrix in Eigure G-6, Appendix G, 
of the NASA SE Handbook. The modified TRL Workbook is as Ref. 10. The modifciations to the TRL Workbook 
are summarized below. 

A. Single Set of Instructions 

The TRL workbook contained instructions for completing the workbook, but these instructions were located in 
two different tabs. The ‘Background and version info’ tab provided some top-level guidelines while the ‘Mission 
TRL Assessment’ tab provided a more explicit set of instructions. In practice, it was found that the information 
given in the two tabs was inconsistent and not easily located, due to neither of the tab labels saying anything about 
‘Instructions.’ The workbook has been modified to contain one tab clearly labeled as “Instructions”. In addition, the 
instructions have been made more explicit by describing exactly what information is needed in each cell of the 
spreadsheet. 

B. Background - Guidance 

The TRL workbook contains a “Background - Guidance” tab which provides an overview of NASA’s best 
business practices related to technology readiness assessment and development. The “Background - Guidance” tab 
has excerpts of best practices from NPR 7120. 5E, NPR 7120.8, and SP-2007-6105 Rev 1 contained within the tab. 

C. Reformatted to Follow Guidance in the NASA SE Handbook 

The overall format of the TRL Worksheet was modified to include columns which can be filled in to indicate the 
TRL earned by various categories for an element. The TRL for that element can be determined by the lowest 
number on that row. A wide text box is provided for a full justification of the numbers entered. An image of the 
blank new workbook is shown in Eigure 3. 
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Product Name 

1 

Product Description 



Opcr»tioB«l EaTiroBaCBt D«scviplioB 

TBD 

DescriplioB 

TBD 




rrr 

1 Description 



l.l 

Subsosiem 1. 1 Name 

1 1 1 

Swbsysttn 1.1 Description 



Coapoaool l.t.1 

Comoonent 1.1.1 Name II II 1 1 

1 1 1 

Comoonent 1.1.1 Descriotion 



HowTecUoloqf 1.1.2a 

New Technoloavf 1.1.2a II II 1 1 

1 1 1 

NewTectiAoloqv 1.1.2e Description 

KcyTechnotogy: 


Development $lttgs 


TRL Jestificetions 


CoBpOBOOl 1.1.3 

Comoonent1.1.3Name II II 1 1 

1 1 1 


Figure 3. Overview of Modified TRL Worksheet. Reformatted to follow guidance in the NASA Systems 
Engineering Handbook. 


D. Product Breakdown Functional Diagram 

The completed 


workbook must now 
be accompanied by 
an hierarchical 

breakdown of the 
hardware products. 

The TRL workbook 
proved to be 
insufficient with the 
early reviews and it 
became apparent that 
additional 

information was 
needed to aid in the 
understanding not 
only of the 
technology item, but 
of the system 
architecture the 

technology would 
eventually be infused 
into. The additional 
supplemental 
material served to 
increase the review 
team’s overall understanding at the system, subsystem, and component levels. 

Contained within the workbook is an example, using the INFLAME instrument, of an hierarchical breakdown. 
This breakdown is consistent with the product breakdown contained within the TRL worksheet. As shown, the 
INFLAME instrument has two systems composed of two spectrometers (LW, SW). Contained within the shortwave 
spectrometer are two subsystems each with a different number of assemblies and components. Additionally for 
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Figure 3. Product Functional Diagram. Include a figure that illustrates the 
breakdown of the system into subsystems and components, corresponds to how the data 
is organized in the TRL Worksheet tab. 



clarity, the TRLs shown and denoted for each level of the product breakdown. This includes using the weakest link 
methodology and the rolling up the TRL to the highest level of the hierarchy. 

Note; the system hierarchical structure can be broken down into many various substructures. No matter how the 
system archeticture is subdivided, it is important to ensure that there is a consistent mapping from the diagram to the 
product breakdown structure used in the TRL worksheet. 


E. TRL Color Coding 

The TRL worksheet has color codes which serves the purpose of providing an 
indication of the technology risk for the new technology and its rollup. Lowest level 
TRLs are denoted with red symbolizing technologies carrying the highest risk. 

Highest level TRLs, TRLs six or higher, are denoted with green symbolizing 
technologies carrying the lowest risk and yellow symbolizing medium risk. The 
five-level risk color codes are shown in Figure 4. 

F. Modiflcations to TRL Worksheet 

As described earlier in Section II, the old workbook contains separate tabs with a series of questions that, when 
answered, will guide the assessor to the appropriate TRL level for that item. In practice, it was found that the 
questions, although helpful, were not structured in a way that would provide confidence in the TRL level thus 
derived. The new workbook still uses the rows of the assessment matrix to provide the framework for the 
hierarchical breakdown of the hardware and software products within the arrangement of the program/project 
product breakdown structure. The columns of the matrix have been enhanced to include more classifiers used to 
determine the maturity of a particular technology, -i.e., one category or set of classifiers would assess the 
environment in which testing of the new technology has occurred. 

There are five categories used to characterize a maturity level for each of the key technologies contained within 
the system. The five categories are: Previous Technology Development, Model Fidelity and Unit Description, 
Environment, Performance/Function Verification, and Verification Documentation. An illustration of the column 
categories are shown below in Figure 5. 






Red (TRL < 2) 
Orange (TRL 3) 
Tan (TRL 4) 
Yellow (TRL 5) 
Green (TRL >6) 


Figure 4. TRL Color 
Code. 



Figure 5. TRL Worksheet Column Categories. The columns indicate what information goes into determining 
a TRL. A TRL number in each group of columns leads to an overall TRL. 


1. Previous Technology Development 

Since many early technology developments within NASA Earth science are managed by ESTO, the TRL 
worksheet captures previous ESTO readiness level assessments. The past ESTO project documentation includes the 
technology maturity at the beginning of the funded project and the exit TRL and at project completion along with 
the project completion date. 

2. Model Fidelity and Unit Description 

Both the model fidelity and unit description (form, fit, and function) provides a means of quantifying the 
hardware or software fidelity based on its final configuration both in terms of final design intent and performance. 
The fidelity of the hardware/software can often take on different meaning depending on the assessors past 
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experience"^. The TRL workbook contains a tab with hardware/software definitions, from NPR 7120.8, Appendix J, 
for the different levels of product fidelity to be used for the workbook. 

3. Environment 

The technology readiness assessment depends on the specifics of the intended system application. Identification 
of the operation environment is required to be documented at the beginning of the worksheet thus the intended 
operational environment must be known. Additionally, along with the operation environment, a description of the 
relevant environment of the system must be specified. For example, in the INFLAME example, the operational 
environment is specified as a 705 km, sun-synchronous orbit with a three year mission duration. For the relevant 
environment, the critical subset of final product performance, is specified as vibration testing of the shortwave 
optomechanical assemblies and thermal vacuum testing of the electronic subsystem assemblies. Knowledge of both 
the operational and relevant environment is needed to conduct an effective TRL assessment. 

The environment in which the new technology was tested is expected to be filled out for each of the new 
technology items. This will aid in determining the degree of similarity between the new technology environmental 
testing and to the environment in which the technology will be used in operation. It is this degree of similarity that is 
one factor that helps determine the readiness level of the new technology. 

4. Performance/Function Verification 

New technologies have performance and functional requirements that they must meet to ensure full system 
capability. The degree to which the required level of performance is achieved aids in determining the readiness level 
of the new technology. 

5. Verification Documentation 

Verification documentation generated by technology development activities must be identified to support TRL 
determination. Documentation provides evidence that the new technology meets specified requirements for intended 
use. Documentation serves as an exit criteria for each of the TRL levels in NASA’s TRL definitions and must be 
completed before entering the next higher TRL level. 

G. Modifications to the Example 

The workbook contained a tab labeled ‘INFLAME Example’ as shown in Figure 2. This example was intended 
to aid in completing the workbook by giving examples of the information to be entered into each row and column. In 
practice, it was determined that a more comprehensive example will provide significantly better guidance to the 
assessor by addressing the many nuances that can be encountered in performing a TRL assessment. Eor future use, 
the workbook example has undergone substantial enhancement to make it more illustrative of all of the instructions, 
and updated to match the reformatted TRL Worksheet. 

H. Modifications to the Training 

Since the SEWG determined that the ‘ad hoc’ training provided was considered inadequate, improvements have 
been developed. Eirst, the improvements to the instructions discussed above will make user training much easier. 
Secondly, a PowerPoint^M tutorial was developed to help guide the user in completing the TRL worksheet. These 
improvements should make completion of the worksheet less confusing for the user. Euture training sessions for 
those unfamiliar with the TRL workbook will be conducted with sufficient time allowed for effective exchange of 
information. 


VI. Summary 

The NASA TRL assessment process has been carried out on several Earth science instrument concepts using an 
assessment workbook. The workbook used for those assessments was found to be adequate and compatible with the 
NASA TRL assessment methodology, but in need of improvement. The SEWG has incorporated the findings and 
recommendations into a revised TRL review process and workbook, with the goal of having a much improved 
process that can be used for the next set of future TRL reviews. A copy can be obtained via the web site given in the 
references, or by contacting an author. 
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